[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

ymao1 · 2024-11-19T17:27:07Z

Resolves https://github.com/elastic/response-ops-team/issues/251

Note

This PR includes some saved object schema changes that I will pull out into their own separate PR in order to perform an intermediate release. I wanted to make sure all the schema changes made sense in the overall context of the PR before opening those separate PRs.

Update: PR for intermediate release here: #203184 (Merged)

Summary

Adds ability to run actions for backfill rule runs.

Updates schedule backfill API to accept run_actions parameter to specify whether to run actions for backfill.
Schedule API accepts any action where frequency.notifyWhen === 'onActiveAlert'. If a rule has multiple actions where some are onActiveAlert and some are onThrottleInterval, the invalid actions will be stripped and a warning returned in the schedule response but valid actions will be scheduled.
Connector IDs are extracted and stored as references in the ad hoc run params saved object
Any actions that result from a backfill task run are scheduled as low priority tasks

To Verify

Create a detection rule. Make sure you have some past data that the rule can run over in order to generate actions. Make sure you add actions to the rule. For testing, I added some conditional actions so I could see actions running only on backfill runs using kibana.alert.rule.execution.type: "manual". Create actions with and without summaries.
Schedule a backfill either directly via the API or using the detection UI. Verify that actions are run for the backfill runs that generate alerts.

kibanamachine · 2024-11-21T03:34:56Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7453

[✅] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 200/200 tests passed.

see run history

ymao1 · 2024-11-25T16:52:22Z

x-pack/plugins/actions/server/create_execute_function.ts

@@ -171,15 +177,17 @@ export function createBulkExecutionEnqueuerFunction({
          executionId: actionToExecute.executionId,
          consumer: actionToExecute.consumer,
          relatedSavedObjects: relatedSavedObjectWithRefs,
+          ...(actionToExecute.apiKeyId ? { apiKeyId: actionToExecute.apiKeyId } : {}),


Storing apiKeyId in the action_task_param saved object so that the API key invalidation task can query for any scheduled actions that are using invalidated API keys and wait before performing invalidation.

elasticmachine · 2024-11-25T21:19:51Z

Pinging @elastic/response-ops (Team:ResponseOps)

mikecote · 2024-11-27T16:53:23Z

x-pack/plugins/task_manager/server/queries/mark_available_tasks_as_claimed.ts

+          if (doc['task.priority'].size() != 0) {
+            return doc['task.priority'].value;
+          } else if (params.priority_map.containsKey(taskType)) {


Moving the conversation from #199218 (comment) here, sorry for the delay!

Either would work! The way it currently is, we would sort by priority=descending which makes more sense in my head but definitely an argument for the reverse

Hmm, is it easy to check what other queueing products like RabbitMQ, Kafka, etc use for priority order? We can see if they do asc or desc and follow suit.

For RabbitMQ: Priorities between 1 and 255 are supported, Larger numbers indicate higher priority.

For ActiveMQ: The full range of priority values (0-9) are supported by the [JDBC](https://activemq.apache.org/components/classic/documentation/jdbc-support) message store. For [KahaDB](https://activemq.apache.org/components/classic/documentation/Persistence/kahadb) three priority categories are supported, Low (< 4), Default (= 4) and High (> 4).

For Redis, priority can be implemented using sorted sets, using the score to represent task priority. For sorted sets, If B and A are two elements with a different score, then A > B if A.score is > B.score.

Kafka and SQS do not support event/message level priority. Instead they both recommend adding different topics and reading from them in priority level (read from high priority topic/queue first, then fallback to medium priority if nothing in the high priority topic/queue, etc).

Seems like in general, higher value = higher priority?

Thanks for doing the research, it sounds in line with the way we have it implemented today so we are good 👍 ❤️

…ill-actions

ymao1 · 2024-12-12T20:22:38Z

One thing I'm wondering about. To figure out if the action was from a backfill, I used an original_event time value, which seemed to work. I think if I was going to use this, I'd want to have a way to know if this is a backfill, to put some kind of warning in my mustache template like
{{#isBackFill}}**Note: this was generated from a backfill**
{{/isBackFill}}
We can obviously do this later, but I was a little curious and haven't figured out where that would go, in our code ...

We have a field in our alert document: kibana.alert.rule.execution.type which can be either "manual" or "scheduled". Seems like we could do conditional actions to add 2 actions that print different messages based on whether the run was from normal execution or a backfill.

pmuellr · 2024-12-12T23:44:03Z

We have a field in our alert document: kibana.alert.rule.execution.type which can be either "manual" or "scheduled". Seems like we could do conditional actions to add 2 actions that print different messages based on whether the run was from normal execution or a backfill.

Does "manual" already have a use? Can we use a new value "backfill"?

ymao1 · 2024-12-13T01:52:31Z

Does "manual" already have a use? Can we use a new value "backfill"?

In this case manual = backfill. The detections UI calls them manual rule runs so they wanted to set the value this way so it shows up consistently in their UI.

pmuellr · 2024-12-13T21:34:56Z

Seems like we could do conditional actions to add 2 actions that print different messages based on whether the run was from normal execution or a backfill.

Ya, that should work.

I was actually thinking about context variables I guess, and not alert fields. Still think we can just do this later, and we may get some requirements for it as well ...

ymao1 · 2025-01-07T21:38:10Z

@elasticmachine merge upstream

…iate release (elastic#203184) ## Summary This PR contains just the schema changes required to support backfill actions. This is meant for an intermediate release and then the full PR: elastic#200784 will follow after that. --------- Co-authored-by: Elastic Machine <[email protected]>

ymao1 · 2025-01-16T15:01:08Z

@elasticmachine merge upstream

ymao1 · 2025-01-17T13:15:44Z

@elasticmachine merge upstream

ymao1 · 2025-01-17T14:36:45Z

@elasticmachine merge upstream

ymao1 · 2025-01-20T13:10:42Z

@elasticmachine merge upstream

elasticmachine · 2025-01-20T14:17:09Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 589feb5
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-200784-589feb5c9a85
Security Deployment

Failed CI Steps

FTR Configs #21

Test Failures

[job] [logs] FTR Configs #21 / console app console autocomplete feature Autocomplete behavior JSON autocompletion with placeholder fields

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`taskManager`	61	62	+1

Unknown metric groups

API count

id	before	after	diff
`taskManager`	104	105	+1

ESLint disabled line counts

id	before	after	diff
`@kbn/test-suites-xpack`	737	738	+1

Total ESLint disabled count

id	before	after	diff
`@kbn/test-suites-xpack`	762	763	+1

History

💛 Build #268123 was flaky 322b099
💔 Build #268083 failed 1e7aff1
💛 Build #267783 was flaky 7b14134
💛 Build #264963 was flaky f9e6478

cc @ymao1

kibanamachine · 2025-01-20T15:03:51Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12870773918

kibanamachine · 2025-01-20T15:08:52Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 200784

Questions ?

Please refer to the Backport tool documentation

ymao1 · 2025-01-20T23:34:58Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ule runs (elastic#200784) Resolves elastic/response-ops-team#251 ## Note This PR includes some saved object schema changes that I will pull out into their own separate PR in order to perform an intermediate release. I wanted to make sure all the schema changes made sense in the overall context of the PR before opening those separate PRs. Update: PR for intermediate release here: elastic#203184 (Merged) ## Summary Adds ability to run actions for backfill rule runs. - Updates schedule backfill API to accept `run_actions` parameter to specify whether to run actions for backfill. - Schedule API accepts any action where `frequency.notifyWhen === 'onActiveAlert'`. If a rule has multiple actions where some are `onActiveAlert` and some are `onThrottleInterval`, the invalid actions will be stripped and a warning returned in the schedule response but valid actions will be scheduled. - Connector IDs are extracted and stored as references in the ad hoc run params saved object - Any actions that result from a backfill task run are scheduled as low priority tasks ## To Verify 1. Create a detection rule. Make sure you have some past data that the rule can run over in order to generate actions. Make sure you add actions to the rule. For testing, I added some conditional actions so I could see actions running only on backfill runs using `kibana.alert.rule.execution.type: "manual"`. Create actions with and without summaries. 2. Schedule a backfill either directly via the API or using the detection UI. Verify that actions are run for the backfill runs that generate alerts. --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 075806b) # Conflicts: # x-pack/platform/plugins/shared/actions/server/create_execute_function.ts

…fill rule runs (#200784) (#207273) # Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Alerting] Adding ability to run actions for backfill rule runs (#200784)](#200784)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <[email protected]>

…ule runs (elastic#200784) Resolves elastic/response-ops-team#251 ## Note This PR includes some saved object schema changes that I will pull out into their own separate PR in order to perform an intermediate release. I wanted to make sure all the schema changes made sense in the overall context of the PR before opening those separate PRs. Update: PR for intermediate release here: elastic#203184 (Merged) ## Summary Adds ability to run actions for backfill rule runs. - Updates schedule backfill API to accept `run_actions` parameter to specify whether to run actions for backfill. - Schedule API accepts any action where `frequency.notifyWhen === 'onActiveAlert'`. If a rule has multiple actions where some are `onActiveAlert` and some are `onThrottleInterval`, the invalid actions will be stripped and a warning returned in the schedule response but valid actions will be scheduled. - Connector IDs are extracted and stored as references in the ad hoc run params saved object - Any actions that result from a backfill task run are scheduled as low priority tasks ## To Verify 1. Create a detection rule. Make sure you have some past data that the rule can run over in order to generate actions. Make sure you add actions to the rule. For testing, I added some conditional actions so I could see actions running only on backfill runs using `kibana.alert.rule.execution.type: "manual"`. Create actions with and without summaries. 2. Schedule a backfill either directly via the API or using the detection UI. Verify that actions are run for the backfill runs that generate alerts. --------- Co-authored-by: Elastic Machine <[email protected]>

ymao1 force-pushed the backfill-actions branch 8 times, most recently from 07d9365 to 8010638 Compare November 20, 2024 20:40

Adding ability to run actions for backfill rule runs

6a60ecb

ymao1 force-pushed the backfill-actions branch from bf12afa to 6a60ecb Compare November 21, 2024 00:47

Merge branch 'main' of github.com:elastic/kibana into backfill-actions

6176db2

ymao1 changed the title ~~Adding ability to run actions for backfill rule runs~~ [Response Ops][Alerting] Adding ability to run actions for backfill rule runs Nov 25, 2024

Default to true if run_actions not specified

df33d0c

ymao1 commented Nov 25, 2024

View reviewed changes

ymao1 added 3 commits November 25, 2024 12:42

Default to true if run_actions not specified

51124d3

Handling rule level notify when field

2a656bd

Cleanup

56fe8db

ymao1 self-assigned this Nov 25, 2024

ymao1 marked this pull request as ready for review November 25, 2024 21:19

ymao1 requested review from a team as code owners November 25, 2024 21:19

mikecote reviewed Nov 27, 2024

View reviewed changes

ymao1 added 2 commits December 12, 2024 14:09

Fixing test

4a9ebb9

Merge branch 'backfill-actions' of github.com:ymao1/kibana into backf…

0de2c31

…ill-actions

Merge branch 'main' of github.com:elastic/kibana into backfill-actions

bc22bd3

Merge branch 'main' into backfill-actions

f9e6478

Merge branch 'main' into backfill-actions

7b14134

ymao1 added the ci:project-deploy-security Create a Security Serverless Project label Jan 16, 2025

Merge branch 'main' into backfill-actions

1e7aff1

Merge branch 'main' into backfill-actions

322b099

Merge branch 'main' into backfill-actions

589feb5

ymao1 merged commit 075806b into elastic:main Jan 20, 2025
8 checks passed

ymao1 deleted the backfill-actions branch January 20, 2025 15:03

ymao1 mentioned this pull request Jan 20, 2025

[8.x] [Response Ops][Alerting] Adding ability to run actions for backfill rule runs (#200784) #207273

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

ymao1 commented Nov 19, 2024 •

edited by kibanamachine

Loading

kibanamachine commented Nov 21, 2024

ymao1 Nov 25, 2024

elasticmachine commented Nov 25, 2024

mikecote Nov 27, 2024

ymao1 Dec 2, 2024

mikecote Dec 2, 2024

ymao1 commented Dec 12, 2024

pmuellr commented Dec 12, 2024

ymao1 commented Dec 13, 2024

pmuellr commented Dec 13, 2024

ymao1 commented Jan 7, 2025

ymao1 commented Jan 16, 2025

ymao1 commented Jan 17, 2025

ymao1 commented Jan 17, 2025

ymao1 commented Jan 20, 2025

elasticmachine commented Jan 20, 2025 •

edited

Loading

API count

ESLint disabled line counts

Total ESLint disabled count

kibanamachine commented Jan 20, 2025

kibanamachine commented Jan 20, 2025

ymao1 commented Jan 20, 2025

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

Conversation

ymao1 commented Nov 19, 2024 • edited by kibanamachine Loading

Note

Summary

To Verify

kibanamachine commented Nov 21, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7453

ymao1 Nov 25, 2024

Choose a reason for hiding this comment

elasticmachine commented Nov 25, 2024

mikecote Nov 27, 2024

Choose a reason for hiding this comment

ymao1 Dec 2, 2024

Choose a reason for hiding this comment

mikecote Dec 2, 2024

Choose a reason for hiding this comment

ymao1 commented Dec 12, 2024

pmuellr commented Dec 12, 2024

ymao1 commented Dec 13, 2024

pmuellr commented Dec 13, 2024

ymao1 commented Jan 7, 2025

ymao1 commented Jan 16, 2025

ymao1 commented Jan 17, 2025

ymao1 commented Jan 17, 2025

ymao1 commented Jan 20, 2025

elasticmachine commented Jan 20, 2025 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

ESLint disabled line counts

Total ESLint disabled count

History

kibanamachine commented Jan 20, 2025

kibanamachine commented Jan 20, 2025

💔 All backports failed

Manual backport

Questions ?

ymao1 commented Jan 20, 2025

💚 All backports created successfully

Questions ?

ymao1 commented Nov 19, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Jan 20, 2025 •

edited

Loading